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1. Introduction 

Let / be an integral homogeneous polynomial of degree d in n variables. A basic prob- 
lem in Diophantine analytic number theory is to understand the behavior of the integral 
representations of integers m by / as m tends to infinity. 

For each m S N, consider the level variety Vm := {x £ M" : f{x) = m}. For instance, if 
f{xi, ■ ■ ■ , Xn) = + • • • + {n > 3), then Vm is the sphere of radius ^/m centered at the 
origin and the set Vm(^) = Vm H consists of integral vectors the sum of whose squares is 
equal to m. In this case, the asymptotic of #T4rx(^) is well known. For n > 5 the classical 
Hardy-Littlewood circle method applies and for n = 4 the Kloosterman sum method works. 
For n = 3, Linnik gave a conditional answer and later Iwaniec gave a complete answer (see 
[Sa], [Iw]). 

In the case when Vm is non-compact, the number may be infinite. In this case 

one asks if there exists an asymptotic density for as m — > oo . To be more precise, for 

a compact subset of Vi , set 

(1.1) Nmif, n) := ^Vmi-L) n mi/'^f). 

Or equivalently, 

7V„(/,0) = #y„(z)nM+fi 

where M+Q is the radial cone {x ^V : tx £ Q for some t G M"*"}. The question we study in 
this paper is if there exists a sequence of numbers cOm, independent of the compact subset fi, 
such that 

Nmif, ^) ~m-^oo UJm ' Vol(O). 

This formulation is basically due to Linnik ([Li2], see also [Sa]). 

The only known general method is the Hardy-Littlewood circle method. However for this 
method to work, the number of variables needs to be much larger than the degree of the 
polynomial in general. 

In this paper, we focus on the polynomials that are invariant under an action of a semisim- 
ple real algebraic group. In such cases, the level varieties admit actions of a semisimple 
algebraic group and the dynamics of such groups plays a crucial role in understanding this 
question. In particular, when Vi is a homogeneous space of a semisimple real algebraic group 
with the stabilizer being generated by unipotent flows, we are able to use a well developed 
theory of unipotent flows on a homogeneous space of a Lie group. 

The first author is partiaUy supported by Packard foundation. 

The second author partialiy supported by NSF grants DMS 0070544 and DMS 0333397. 

1 



2 



ALEX ESKIN AND HEE OH 



We formulate our main results for a family of varieties Vm which are more general than 
level sets of a given polynomial. Let V be a finite dimensional real vector space with a Z- 
structure. Let G be a semisimple real algebraic group defined over Q and let p : G ^ GL{V) 
be a Q-rational representation of G. Fix a non-zero vector vq E 1^(Z) such that vqG is Zariski 
closed and a sequence {Am G : m G N} of strictly increasing numbers. We set 

Vm := XmVoG for each m G N. 

Denote by Ay be the collection of all arithmetic subgroups F C G(Q) of G such that 
V{Z)r C V{Z). Since p is defined over Q, Ay is non-empty. Consider a sequence {Cm C 
VmC^) : m G N} of F-invariant subsets of VmC^) for some F G Ay- For a compact subset Cl 
of Vi, we define 

where pr : Vm — > Vi denotes the radial projection given by pr(x) = AiA~^x. 

Let H denote the stabilizer of vq in G. The notation and G^ denote the identity 
components of H and G respectively. In the rest of the introduction, we assume that H is 
semisimple without compact factors. 

Theorem 1.2. Suppose that is a maximal connected closed subgroup of G^. Let {Om C 
V^(Z)} be a sequence of T-invariant subsets for some F G Ay, for example, Om = Vm{'^)- 
Suppose that for each ruQ G N, 

(1.3) #{mGN:pr(Om„)=pr(0™)}<oo. 

Fix a G-invariant Borel measure p onVi. Then for any compact subset Q ofVi with boundary 
of measure 0, 

(1.4) N{Om,n) (Om) ■ pm 

where io{Om) is given below lll.b]) . 

An immediate consequence of the above theorem is that the sequence {pr{Om)} of projec- 
tions is equidistributed on Vi as m ^ oo, since it follows from 1)1.4^ that for any two compact 
subsets r^i and of Vi with boundary measure 0, 

Nm{Om,^l) /i(^l) 
Nm{Om,^2) Pi^2)' 

The condition H1.3|) says that there is no infinite sequence of varieties V^. where all the 
integer points in Omi are simply the radial dilations of a fixed Omi ■ Clearly this is a necessary 
condition for the conclusion of the above theorem. We remark that the condition l\l.'d\i is also 
equivalent to saying limm-*oc^{Om) = oo (see Lemma . 

In order to give the formula of the number uj{Om) in Theorem ll.2[ let pc and pn denote 
Haar measures on G and H respectively so that the triple {p, pc, Ph) are topologically 
compatible in the sense of Weil [We], namely, for any continuous function f on G with 
compact support, 

(1.5) / fdpG{g)= [ dp{-g) [ f{hg)dpH{h) 

JG Jh\g jh 

where /i, by slight abuse of notation, denotes the G-invariant measure on H\G pulled back 
from p on Vi via the identification of H\G with Vi by Hg i— > v^g. 
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The number u;{Om) is defined as follows: for any T £ Ay preserving each 



(1.6) a;(a 



/^G(r\G) 



where the sum is taken over the set of disjoint F-orbits in Om and i^g G G is an element 
such that = pr(^)g^. Under our assumption, lo(P„^ < oo, and moreover uj{Om) does not 
depend on the choice of T in Ay preserving Om, justifying our notation. This is shown in 
[Oh] for Om = VmC^) but the same proof works for any Om- For the case of Om = VmC^), 
we simply write 

(1.7) LOm = Oj{Vmm. 

Note also that the product Lv{Om)^^^{^) is independent of the choice of compatible measures 
{fi, fiH, IJ-g)- For this reason, we sometimes write u;{Om) ■ vol(J7) in what follows. 

We remark that Theorem 11.21 applies to the cases where Vi is an affine symmetric space, 
i.e., H is the set of fixed points of an involution of G, with additional assumptions that G 
is Q-simple and that H is semisimple without compact factors. It is so since is then a 
maximal connected closed subgroup of (cf. [Bo, Lemma 8.0]). 

We present some examples which follow from Theorem 11.21 for the case Om = ymC^)- 
See (|1.1|) for the notation Nm{f,^)- The following three theorems are proven in sectional 
where we realize each / as an invariant polynomial of a certain representation of a semisimple 
algebraic group. Once we do that, the number ujm is defined as in (|1.6j) and (|1.7|) . 

Theorem 1.8. Let n > 2 and Det denote the determinant polynomial on the vector space 
Mji(M) of n X n matrices. For any compact subset ofVi = {X G Mn(M) : Det(X) = 1} of 
boundary of measure 0, we have 

Nm(Det, Q) ~m^oo l^m • Vol (Q). 

Theorem II. 81 was proven first by Linnik [Lil] (for n = 2) and by Linnik and Skubenko (for 
n > 3) [LS] in the early sixties using methods in analytic number theory. A different proof 
was also given using methods based on Hecke operators (see [Sa], [COU] and [GO]). 

Let V be the subspace of M2n(IR) consisting of the skew-symmetric matrices, i.e., V = 
{X E M2n(K) : X^ = —X} in > 2) and consider the Pfaffian as a polynomial on V defined 
by Pff(X)2 = Det(X) and 

where /„ denotes the identity matrix of order n. 

Theorem 1.9. Let n > 2. For any compact subset Q, of Vi = {X G V : Pff(X) = 1} of 
boundary of measure 0, we have 

A''m(Pff , 0,) ~m->oo t^m • Vol (0,). 

An integer m is called a fundamental discriminant if and only if m is either a square free 
integer congruent to 1 mod 4, or 4 times of a square free integer which is congruent to 2 or 
3 mod 4. 
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Theorem 1.10. Let Q he an integral quadratic form of signature {r,s) where r + s > 4, 
r > 2 and s > 1. For any compact subset 0, ofVi = {X G : Q{X) = 1} of boundary of 

measure 0, 

Nm{Q, fl) ~ • vol (0,) 
as m ^ oo along the fundamental discriminants. 

In fact, the Hardy-Littlewood circle method together with the Kloosterman sum method 
(needed for r + s = 4) (cf. [Va], [Es]) also gives an asymptotic density in Theorem II .101 in the 
form of product of local densities. By comparing the two different forms of the asymptotic 
for Nm{Q,^), one obtains a new proof of Siegel mass formula for quadratic forms (see [Oh]) 
in the same spirit of the work of Eskin, Rudnick and Sarnak [ERS]. The above theorem is 
still true for the case of r + s = 3 and rs > though our method does not apply. This follows 
from a theorem of Duke [Du] . 

More generally, a natural question is whether the asymptotic uJm • vol(r2) in Theorem 11.21 
coincides with the heuristics predicted by the Hardy-Littlewood circle method. It is shown 
in [Oh] that this is true in many cases but not always. 

To state our main result in a more general setting without the maximality assumption on 
H, we define the following: the notation Z(H^) denotes the centralizer of in G. 

Definition 1.11. A sequence {Om C VmC^)} is called focused if, for every T £ Ay pre- 
serving {Om\, there exist a proper connected closed subgroup L of in which L n F is a 
Zariski dense lattice, g £ G with g^^H^g C L and a compact subset C C voZ{H^)gL such 
that for every compact subset Q C Vi, there exists 7^2 G E with 

(1.12) hmsup ^ — — > 0. 

m— >oo ^\^m) 

In the above and also in the rest of the paper, the notation Yl^rcOm ™6ans that the sum 
is taken over the set of disjoint E-orbits contained in Om- 

Theorem 1.13. Suppose that {Om ■ m E N} is not focused. Then for any compact subset O 
of Vi with boundary of measure 0, 

(Om) -yoli^l). 

Note that in the case when Om consists of one F-orbit, the focusing of {Om} implies 
that for every given compact subset of Vi, there exists an infinite sequence rui such that 
pr(OmJ n lies completely inside a proper subvariety of Vi. In general, if {Om} is focused, 
one expects in view of Theorem II . 131 that a positive proportion of points in pr(OmJ H lies 
inside a proper subvariety of Vi of the form voZ{H)gL^ for some 7 S F and for some proper 
subgroup L of G'^. 

Assuming H connected for simplicity, we explain some of schemes in the proofs. Eor 
each ^E C Om, consider the i?-invariant measure i^^r on F\G supported on the closed orbit 

T\Tg(^H with the total measure a;(^r). Let am denote the averaging measure ^^^('0'")'^^'" 
on T\G. We first observe that the equidistribution of pr{Om) (Theorem I1.13P follows if am 
converges to the G- invariant probability measure on T\G as m — > 00 fFroDosition l2.2() . We 
then show that under the non- focusing assumption of {Om}, the /f-orbits T\Tg^H which 
stay outside a given compact subset G as well as the i7-orbits T\Tg^H which stay inside 
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T\TLg^ for some proper subgroup L of G are ignorable, in considering the weak hmits of dm- 
Then by applying main ergodic results of Dani and Margulis in [DMl-2], we show that each 
i?-orbit T\Tg^H is getting longer and longer and moreover uniformly distributed on T\G as 
m ^ oo; hence the average measure am tends to the Haar measure on r\G as m — > oo. 

In particular in the case where the subgroup H is a maximal closed subgroup of G^, the 
equidistribution of {pv{Om)} is a consequence of the phenomenon that for any sequence of 
non-repeated individual F-orbits {pr(^)r C pr(C'm)}, the corresponding sequence {r\r(7^i/} 
of i?-orbits is uniformly distributed on r\G and hence pr(^)r is equidistributed on Vi as 
m — > oo. 

Remark Gan and Oh [GO] showed that for any invariant polynomial / in the above setting 
but with a more general H (not necessarily semisimple) , if V^q (Z) contains at least one integer 
point for some do G N, then there exist explicitly computable constants c and r depending 
only on G, p and deg (/) such that pr(T4(iom''(^)) becomes dense on Vi in a strong sense as 
m — > oo. In particular it follows that the limit supreme in (|1.12l) is strictly less than 1. 
Acknowledgment The second named author would like to thank Peter Sarnak for suggesting 
this problem as well as for helpful discussions. 



2. Measure theoretic formulation of a counting problem 

Let G be a real semisimple algebraic group defined over Q. This means that there exists 
a connected semisimple algebraic group G defined over Q such that G is a closed subgroup 
of G(M) containing the identity component G(M)°. Let p : G ^ GIj{V) be a Q-rational 
representation for a finite dimensional real vector space V defined over Z. Fix a non-zero 
vector vq £ V such that vqG is Zariski closed and a sequence {A^ : m G N} of strictly 
increasing positive numbers. For each m G N, we set 

Vm = XmVoG. 

Let H denote the stabilizer of vq in G. Since p is rational, H is a real algebraic subgroup 
of G. In particular, the identity component is a finite index normal subgroup of H. 
Assume that has no non-trivial M-character. Let pc denote a Haar measure on G. Since 
both G and H are unimodular, we may choose a G-invariant Borel measure p on Vi and an 
iif-invariant measure pn on H so that the triple {p, pc Ph) is compatible in the sense of 

Let Ay be defined as in the introduction, and let F G Ay- Since G acts transitively on 
Vi, for any ^ G Vm, there exists g^ G G such that vq = pr(^)<^^. The choice of g^ is unique 
only up to modulo H. If denotes the stabilizer of in G for ^ G Vm{^), then is a 
Q-subgroup of G and = g^Hg^^ . Therefore has no non-trivial Q-character and hence 
by a theorem of Borel and Harish-Chandra, n F is a lattice in H^. Hence 

■= MTAG) ^ 

Observe that the definition of a;(.^F) depends only on the F-orbit not on its representative. 

Let {Om C Kn(Z)} be a sequence of non-empty F-invariant subsets of 14n(Z) for some 
F G Av- Since each Vm is Zariski closed, by a theorem of Borel and Harish-Chandra, the 
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number of F-orbits in Kn(Z) and hence in Om is finite. Hence 

u;{Om):= ^(Cr)<oo- 
Lemma 2.1 (Oh). The number to {Om) is independent of the choice o/F G Av preserving 

Om. 

The space V{T\G) of the probabihty measures on F\G is equipped with the weak*- 
topology. Now fix any F G Av which preserves each Om- For each F-orbit ^F C Om, let u^-p 
denote the unique i7-invariant measure on F\G supported on the closed orbit T\Tg^H and 
with the total measure given by u;(^F). Hence 

Here is a main proposition suggested by Sarnak which translates the counting problem to 
the question of whether the weak- limits of the above measures are G-invariant. 

Proposition 2.2. // 

m^oo Lj{Om) ^f^^ Mg(F\G) 

then for any compact subset Q of Vi with boundary of measure 0, 



N{Om, 0) ^m-.oo CuiOm) ■ /x(J^). 

Proof. Without loss of generality, we assume iJ,G(r\G) = 1. Let ip be any continuous function 
with compact support on H\G = Vi. Define a function as follows: for each g E G 



Since is left F-invariant, it may be considered as a function on F\G. Let -0 be a continuous 
function on F\G with compact support. Note that 
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= E / ./.(pr(e)5ci)V'(5cOdMG(i) 

= E / '/'(^oS') / '4'{9ihg)d^iH{h) d^i{g) 

Consider a function ^/^^ on r\G defined by V'g(s) '4'{s9)- Then 



Hence by the assumption, 



r\G Jv\G 



^™ //o A E / ^{sg) dvi:{s) = I ijdfiG. 

i^oo Uj{Om) Js€T\G Jr\G 



Now by the Lebesgue dominated convergence theorem 



If follows that 



hm {F^,i;) = / <^>{vog)d^i{g) ■ / V^d/ic- 
IH\G Jr\G 



(2.3) lim_(F™,V') = M^^)- / V'd^. 



^G 



r\G 



where xn denotes the characteristic function of fi. 

Fix e > 0. Let be a symmetric neighborhood of e in G such that 

-n-)<e 

where ^if = UueuM''^ = ^ueUc^u. Then for all g £ C/^, 

(2-4) I'xnj9)<FTni^)<FTnj9)- 

Let V'e be a non-negative continuous function on r\G with support in Ue and J-p\Q V'e dfj^c = 
1. Integrating (|2.4() against V'e now gives 

Since both sides tend to respectively as m — > oo by ()2.3() and e > is arbitrary, we 

have 

F™(e)^//(17) asm^oo. 
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Since 




this proves the claim. 



□ 



3. Asymptotic behavior of unipotent flows 



We recall the following fundamental result of Dani and Margulis. 

Theorem 3.1 (DM2, Theorem 6.1). Let G he a connected Lie group and V a lattice in G. 

Given a compact subset C C r\G and an e > 0, there exists a compact set K C r\G such 
that the following holds: for any x E C, any unipotent one-parameter subgroup {u{t)} of G, 
and any T > 0, 



where \ ■ \ denotes the Lebesgue measure on R. 

Let G be a connected semisimple real algebraic group defined over Q, and H a connected 

semisimple real algebraic subgroup of G. Let T C G'(Q) be an arithmetic subgroup of G. 
The results in this section have meanings only when r\G is non-compact, which we assume. 
Consider the one point compactification r\GU {cx)} of r\G. The space V{T\GU {oo}) of the 
probability measures on r\G U {oo} equipped with the wcak*-topology is weak* compact. 

Let {grn G G} be a sequence such that QmHg^^ is a Q-subgroup of G for each m. By a 
theorem of Borel and Harish-Chandra [BH], it follows that g^Tgm n is a lattice in H. 
Hence each V\rg^H is closed in r\G (cf. [Rag]) and there exists the unique i^-invariant 
probability measure /x^ in r\G supported on r\r^^iJ. 

Proposition 3.2. Assume either that H has no compact factors or that gmHg^ is Q-simple 

for each m. Then the following are equivalent: 

(1) There exists a compact subset C of T\G such that 

^\rgmH n (7 7^ for all sufficiently large m £ N. 

(2) Every weak limit of {fJ-m} ^'^ V{T\G U {oo}) is supported on r\G. 

Proof. Assume that (1) is true. Without loss of generality, we may assume that T\Tgm G C 
for all m G N. Let denote the unique maximal connected normal closed subgroup of H 
without compact factors. Let U = {u{t)} be a unipotent one parameter subgroup in not 
contained in any proper normal subgroup of Hj^. Such a subgroup exists (see for example, 
[MS, Lemma 2.3]). Under our assumption, cither H = Hj\f or g^Tgm H if is an irreducible 
lattice in H. Hence it follows from Moore's ergodicity theorem (cf. Theorem 2.1 in [BM]) 
that U acts ergodically with respect to each fim- Moreover by the Birkhoff ergodic theorem 
(cf. [BM]), for almost all h e H, r\r gjnhu{t) is uniformly distributed on r\G with respect 
to firn, i-e., for any / G Cc{r\G), 



l^oo 1 Jq J 

Therefore we may assume that for each m G N, there exists hm G H such that T\rgmhm G C 
and Tgfnhmu{t) is uniformly distributed on r\G with respect to /x^. For any given e > 0, let 



\{t e [0,T] : xu{t) e K}\ > (1 - e)T 
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K he a compact subset of T\G as in Theorem Kill with respect to C. Then for each m G N, 

fJ-miK) > 1 - e. 

Therefore fJ-{K) > 1 — e for any weak hmit /x of {/Um}- Since e > is arbitrary, we have 
li{T\G) = 1, proving that (2) holds. 

Now suppose that (1) fails. First write T\G as U^-^Ci where Ci are compact subsets such 
that Ci C Cj+i for all i. Then for each i, there exists such that 

r\r<7„,i?nQ = 0. 

Since Ci is increasing, we have 

(3.3) T\TgmjHn Ci = $ for all j > i. 

This implies that any weak limit of {//mi} cannot be supported on r\G, for if so, then for 
some iQ, f^miiCig) > 1/2 for infinitely many i. This is contradiction to (|3.3() . Hence (2) 
implies (1). □ 

Let {Dm C H\G} be a sequence of a finitely many union of T-orbits. For each H\Hg G 
Dm, we assume that gHg~^ is a Q-subgroup of G. For each F-orbit r/F C Dm, set 

w(r/F) := fiG(T\G) ^(^m) — 2^ ^{vn- 

Here grj G G is such that ?? = H\Hgrj. 

Let z/,yr denote the //-invariant measure on F\G supported on T\TgjjH with the total 
measure given by uj(rjT). Define an //-invariant probability measure dm on T\G: 

1 



The notation //jy denotes the unique maximal connected normal closed subgroup of H 
without compact factors. 

Proposition 3.4. Assume either that H has no compact factors or that g^^Hg^^ is Q-simple 
for any rj £ Dm- Suppose that g^Hj\ig~^ is not contained in any proper Q-parabolic subgroup 
of G for any rj G UmDm- Then any weak limit of {am : m G N} in P(F\GU{oo}) is supported 
in T\G. 

Proof. Without loss of generality we may assume that {cTm} converges in ViT\G U {oo}). It 
suffices to show that, for any e > 0, there exists a compact subset K C V\G such that 

am{K) > 1 — e for all sufficiently large m. 

Assume not; then for any compact subset K C F\G, (after going to a subsequence) there 
exists rjm G Dm such that 

Urj^riK) < (1 — e)Lj{7]mT) for each m G N. 

Let U = {u(t)} be a unipotent one-parameter subgroup of //at as in the proof of previous 
proposition. Let R be the set of G // such that T\Tgjj^hu{t) is uniformly distributed in 
T\Tg^^H with respect to the probability measure r) ^Vmr- Then R has the full measure 
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in H (see the proof of the previous proposition). Fix any h & R. Then for each m G N, there 
exists Tm > (depending on h) such that 

^\{te [0,T] : T\Tg,,^hu{t) e K}\ < I - e/2 

for all T > Tm where | • | denotes the Lebesgue measure on M. Applying a theorem of Dani 
and Margulis [DM1, Theorem 2] (see also [EMSl]), we obtain that for any given am > 
with limjn^ooCtm = 0) after passing to a subsequence, there exist a proper parabolic Q- 
subgroup P of G, a non-zero vector q G A^Lie(W)(Q) {W being the unipotent radical of P 
and k = dim(VF)) and a sequence {jmih) G T} such that for all m G N and t > 0, 

where the action is through the k-ih exterior of the adjoint representation of G on a'^ Lie(G). 
Since u{t) acts as a unipotent one-parameter subgroup on A" Lie(G) and any orbit of a 
unipotent one-parameter subgroup is unbounded except for a fixed point, it follows that for 
all < t < DO and for all m G N, 

Hence 

C^C(7^5r,„/i)"'P(7^5,./i), 
since the latter group contains the stabilizer of the vector q-Jmgr]mh- Hence we have shown 
that for almost all h G H and for any m G N, 

huh-^ c {7';^9r,j~'p{iigvJ 

for some 7^ G T. Since F is countable, it follows that for each m G N, there exist an element 
7m G F and a subset Sm <Z H oi positive measure such that 

hUh~^ C {7m9r]m)~^P{lm9vm) for all h G Sm- 

Since the set 

{he H : hUh~^ C (7m5'r?„)~^-P(7m5'»;m)} 
is a real analytic submanifold of H with a positive measure, it is indeed equal to H. Hence 

hUh~^ C ilmgrimr^Pilmgrim) ^^T all h£ H. 

Since U is not contained in any proper normal subgroup of if at, it follows that 

This contradicts the assumption since jm~^P"fm is a proper parabolic Q-subgroup of G. □ 

4. Projections of Om and stabilizer subgroups 

We recall the following theorem of Dani and Margulis: let G be any connected Lie group 
and F a discrete subgroup of G. We fix a left invariant Riemannian metric on G. Let M 
be any closed subgroup of G such that M H F is a lattice in M. Then F\FM is a closed 
Riemannian submanifold of F\G and hence it has a right M-invariant Riemannian volume 
form, denoted by V, induced by the Riemannian metric. 
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Theorem 4.1 (DM2, Theorem 5.1). For any c > 0, let Wc be the collection of all closed 
connected subgroups of G such that r\rAf is closed in T\G and V((M n T)\M) < c. Then 
there are only finitely many subgroups of the form M n F with M G Wc- 

We also need the following simple consequence of a theorem of Kazhdan and Margulis 
([KM], [Ra, Theorem 11.8]): 

Lemma 4.2. Let G be a connected linear semisimple Lie group without compact factors. 
There exists a constant c > such that for any discrete subgroup T of G, the co-volume of T 
in G (with respect to a fixed Haar measure on G) is at least c. 

We keep the same notation from section |21 for G, p : G ^ GL{V), F, H, H^, vq, etc. Let 
Om be a F-invariant subset of Kn(Z) for each m. We assume that H is semisimple without 
compact factors. Denote by N{H) the normalizer of H in G. 

Lemma 4.3. Assume that [N{H) : H] < oo and that for each ^ G Om, is not contained 
in any proper Q-parabolic subgroup of G. Then the following are equivalent: 

(1) Suppose that for each mo G N, 

#{m G N : pr(0„J = pr(O^)} < oo. 

(2) limm^oo'^(C'm) = oo. 

Proof. It is easy to see that pr(C'm) = pr(C'A,.) implies uj{Om) = uj{Ok). Hence (2) implies (1). 
Assume now that (2) fails. Then by passing to a subsequence we may assume that Lv{Om) 
is uniformly bounded. By Lemma 14.21 which we may apply since has finite index in H, 
there exists some c > such that 

w(^r) = 1^ > c for ah 4 G UmOm. 

/^G(i \G) 

Since u}{Om) > hm ■ c where hm is the number of disjoint F-orbits in Om, we may also assume 
that hm is constant, say r, for all m, by passing to a subsequence. Now write Om = U[^]^.^m,F. 
It suffices to show that for each 1 < « < r, pr(^mT) is the same set for infinitely many m. 
For, this implies that pr(Om) is the same set for infinitely many m, which contradicts (1). 
Fix 1 < i < r and set E,mi = ?m for simplicity. 

It follows from Propositions 13 . 2l and 13 .41 that there exists a compact subset G of F\G such 
that T\rg^^I{ n C 7^ for all m G N. Hence we may choose g^^ so that {g^^ : m G N} is 
relatively compact. 

On the other hand, if 5g denotes the factor by which the volumes of subsets gets multiplied 
under the transformation h ghg^^ , h £ H, then 

up a uniform constant multiple depending only on the choice of Haar measure hh- Since 
{ffCm : m G N} is relatively compact, sup^ 5g^^ < 00. 
Therefore 

supV{{H^^nr)\Hi:J < sup 6g^^ •supa;(C'„) < 00. 

m mm 

By Theorem 14.11 this implies that H^^ D F are all equal to each other by passing to a 
subsequence. Since H^^ H F is Zariski dense in H^^ by Borel density theorem, it follows that 
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H^^ are all equal to each other, that is, g^^gs^^ ^ ^i-^) ^'^^ '^j ^- Since [N{H) : H] < oo, 
by passing to a subsequence, we have 

Sirl^^k ^ and hence pr(^mr) = pr(^fcr) 

for all m, k. This finishes the proof. □ 

Observe that for compact subsets <zVi, Cq <Z G such that H\HCq = Q, and ^ G VmC^), 

(4.4) iV(^r, r?) = if and only if r\rg^H n T\rCQ^ = 0. 

Proposition 4.5. Consider a sequence {^m G ^(^)} s^tc/i i/iai {pr(^m)} relatively com- 
pact in V\. Suppose that L is a closed subgroup of G containing for all m. such that 
L nT is a lattice in L. Then for any compact subset Q of Vi, there exists a finite subset 
Af7 C r such that for all m, 

w{im)T r\Vt(z pr(e™)(L n r)An. 

Proof. Let VLq be a compact subset of G such that H\HVtQ = i7. Write H as, a disjoint union 
^^^ihiH^ . By the assumption, there exists a choice of {^^^j so that {^^^ G G} is relatively 
compact. Let ili C G be a compact subset which contains {g^^hi : I < i < k, m £ Njf^o- 

Since L n F is a lattice in L, r\rL is closed in r\G [Rag], and this implies easily that 
L\LT is closed in L\G. Since L\LT is a closed countable subset of L\G, it follows from Baire 
category theorem that there exists at least one isolated point. Since T acts transitively on 
L\LT, every point of L\LT is an isolated point. Therefore L\LT is discrete in L\G. Hence 
there exists a finite subset Aq of T such that 

LQi n LF C LAq. 

Note that 

g^Hno n ^ Lg^mi^^=ihi)^o n LT c LQi n LT c LAn. 

Hence 

(4.6) HnonH'^g^^TCg^J^LAn. 

If vqx G pr(^mr) n for a; G G, then 

X = hg^^'j = h'w 

for some h,h' £ H , j £ T and w G r^o- 
Then 

and hence by 1)4. 6() . 

= fi'^is'Ti for some 5 G L and 71 G A^. 
In particular, g = 77^^^ G L n F. Therefore 

X = hg^^gii G Hg^^iL n F)An 

proving 

pr(UF) n C w{U){L n F)Aa. 

□ 
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5. Asymptotic behavior of Om 

We start by recalling the following theorem of Dani and Margulis. For any two closed 
subgroups U and L of a connected Lie group G, set 

X{U,L) := {geG:gU cLg}. 

Theorem 5.1 (DM2, Theorem 3). Let G be a connected Lie group and T a lattice in G. Let 
U = {u{t)} be a unipotent one-parameter subgroup of G and let be a bounded continuous 
function ofT\G. Let K be a compact subset ofT\G and let e > be given. Then there exist 
finitely many proper closed subgroups Li, - ■ ■ ,Lk such that Li HT is a lattice in Li for each 
1 < i < k, and compact subsets Ci, ■ ■ ■ ,Ck of X{U, Li), • • • , X{U, L^) respectively, for which 
the following holds: for any compact subset F d K — u|^^r\rCi, there exists Tq > such 
that for all X e F and T >Tq, 



^ I Tp{xu{t))dt - I i^dfiG 



< €. 



In fact, it is shown in the proof of the above theorem [DM2] that the subgroups Lj can 
be taken so that Ad{Li fl T) is Zariski dense in Ad(Lj) as well where Ad denotes the adjoint 
representation of G. 

We keep the same notation from section 2 for G, p : G — GL(F), F, vq, etc. Let Om 
be a F-invariant subset of VmC^)- As before, we assume that the subgroup H, which is the 
stabilizer of vq, is a semisimple real algebraic subgroup of G without compact factors. Set 
X = T\G. We assume without loss of generality ijlg{X) = 1. For each ^ G ^/^(Z), we denote 
by z^^r the unique iJ-invariant measure on X supported on T\rg^H with the total measure 
given by a;(^F). 

Denote by vr the canonical projection from H^\G to H\G. By the identification of Vi = vqG 
with H\G, we consider pr(0„J as a subset of H\G. Set Om = 7i"~^(pr(Om))- Note that Om 
is F-invariant and has finitely many F-orbits. For each rjT C Om, the notation Uj^r denotes 
the i/'^-invariant measure on F\G supported on T\Tg^H^ with the total measure given by 

, „ n9-ir<,„)\fl°) 
"f"^' MTW 

where 5^ is any element in G such that H^\H^grj = rj. Here fifjo is simply the restriction 
of iJ,H to H^. Note that if 7r(r7iF) = 7r(?72F), then u;(r/iF) = a;(?72F) and uj{r]T) < uj{^T) if 
7r(r7) =pr(^). 

For ^F C Om, it is not hard to check that 

7jrC7r-l(pr(^r)) 

where the sum is taken over the disjoint F-orbits r]T in Om such that 7r(77F) = pr(^F). The 
number sTich F-orbits is clearly bounded by [H : H^]. 

Therefore 

(5.2) ^ i/^r = XI ^'^r and uj{Om) = ^ >^{v^)- 
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Let U = {u{t)} be a unipotent one-parameter subgroup of not contained in any proper 
closed normal subgroup of H^. For given compact subset K C X, e > and a bounded 
continuous function ip on X, let Li and Cj, 1 < i < A; be as in the above theorem, with 
respect to the given triples (iC, e, ■0). Set 

5(i^,V',e) ■.= Kn{utir\m) 

and let G{K, ip, e) denote the complement of S{K, tp, e) inside K. 

Proposition 5.3. Fix a compact subset K of X with a non-empty interior. Suppose that for 
any e > and for any continuous function ip on X with compact support, 

lim ■ = 1. 

Then 

s yZ '^Cr f^G as oo. 

Proof. We set 

Araie) := {r\rr7^/70 : r/F C r\rr7^F0 n C cS(i^, V, e)} 

and 

B„(e) := {r\rgr,H^ : rvr^^F^ n ^(ir, ^, e) / 0}. 

The assumption implies that 
(5.4) 

1-^ ■■ r\rg^g° £ AM} ^ Q. j.^ E{^(^r) : r\rg,ff° e i?^(e)} ^ ^ 



By the Birkhoff ergodic theorem and Moore's ergodicity theorem, U acts ergodically with 



respect to each and the following subset R has the zero co-measure in H^- 



R = {h G : T\Tgrihu{t) is uniformly distributed in T\TgrfH^ w. r. t. — j--^Unr}- 

Let r\rgriH^ G Bm{e). Since Q{K,'ip,e) is open in and i? has co-measure in H^, we 
may assume by a suitable choice for that T\Tgrj G G{K,ip,e) and 

lim ^ I 'ip{T\Tg^u{t))dt = / / V'c^^'r^r- 
Therefore by applying Theorem 15.11 to each singleton F = {T\Tgri}, we obtain that for any 



(5.5) 



ui{r]T) 



X 



1p dVriT- - 1p dflG 



< e. 
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Now 



r\rg„HOGA 

m.\^ } 



X Jx 



ipdur,r - w(??r) / ipdiJ-G 
X Jx 



+ E 

By ()5.5p . the above is again less than or equal to 

(llV'lloo + \mi){Y,{^ivr) ■■ r\rg,H^ e AUe)}) + ei^^i^m ■■ A^avH' G B^e)}). 

By applying Lebesgue dominated convergence theorem, we deduce from (|5.4|) 



lim sup 



1 



L a 



ipdur,r 



< €. 



Since e > is arbitrary, we have for any bounded continuous function ip on X, 

1 



lim 



■ E 



X 



ipdiy^r 



ipdiXG- 



This proves our claim by (|5.2|) . 



□ 



Lemma 5.6 (EMS2, Lemma 5.1). Let G, H and L he connected real algebraic groups such 
that H C L C G. If at least one of G, H, and L is reductive, then X{H,L) is a union of 
finitely many closed double cosets of the form L ■ g ■ Z{H) where g £ X{H,L). 

Proof of Theorem 11.131 Since G^ has a finite index in G and Vi consists of finitely many 
open G^ orbits, it suffices to prove the theorem for each G'^-orbit. Hence we may assume 
that G is connected without loss of generality. Since {Om} is not focused, there exists an 
arithmetic subgroup T C G{Q) which preserves each Om and {Om} is not focused with 
respect to T. 

First, for some compact subset C of Vi, 

E€rco.Mer):pr(e)rnc = 0} 

hm sup ^ — — = 0. 

Note that the same holds for any compact subset of Vi containing C. 

Hence it follows from the observation (|4.4I) that for some relatively compact open subset 
Co of G, we have 

E,rc£)^{^(^r) : T\Tg,H^ n r\rCo = 0} 

(5.7) hmsup — — = 0. 

m— >oo ^\^m) 

Set K = r\(rCo) and K' = r\(rCo). By Propositions El and ESI it suffices to show 
that for any e > and for any bounded continuous function t/; on X, 

, E,rco„V(^?r) :r\r5,i/0ng(i^,^,6) /0} 

lim — — r = 1. 

m^oo Uj{Om) 
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Suppose not. Since the orbits T\TgriH^ disjoint from K' can be ignored by (|5.7|) . it follows 
that there exist a bounded continuous function ip on X and an e > such that 

(5.8) hmsup — '- — — > 

m— >oo '-'-'(L^7Tij 

Let Li and Cj, 1 < i < khe the subgroups and compact subsets in X{U, Li) respectively, used 
in the definition of S{K,Tp,e). Since S{K,Tp,e) is contained in the finite union ujL]^r\rCj, 
there exists 1 < i <k such that (|5.8|) holds with r\rCj in place of S{K,'ip,e). Without loss 
of generality, we assume i = 1. By Lemma 15.61 there exists g £ G such that gll g~^ C Li and 

ErjrcoJ^m-.di^rg^HOnK'cricnL^gZiu))} 

(5.9) limsup — ■ > 0. 

Whenever 

^ TgriH^ nK' C r{Ci n LigZ{U)) 

we may assume that (7,, G Ci n LigZ{U) by replacing rj and g^ by suitable elements in 
7r~^(pr(^r)) and griH^ respectively. We may also assume that the set {h G : g^h £ 
Ci n LigZ{U)} has a positive measure, since K' is open. Note that grih G Ci n LigZ{U) 
implies that hUh~^ C gri~^Ligj^. By a similar argument as in the proof of Proposition 13.41 
it follows that H^cLi, i.e., g^ G X{H^, Li), whenever / TgnH^ n K' C r{Ci n LigZ{U)) 
and 7r(77) G pr(^r). Applying Lemma IHIHl again, we deduce from (|5.9|) that for some go £ G 
such that goH'^gQ^ C Li, 

E,rcoJ^m--9n^CinLmZ{H^)} 
hmsup — — > 0. 

m— >oo t^vL/mJ 

Since grj £ CiD LigoZ{H) and 7r(r7) = pr(^) implies that gr^H^g~^ = C Li, it follows 
from Proposition 14. 51 that for any compact subset of Vi, there exists a finite subset Aq C F 
such that for all g^j £ GiCl Lig^Z^H), 

tt{i-i)T nQc {Hgr,-^){Li n T)Kn C H{LigoZ{H) n Co)-\Li n r)Af,. 
Hence we have shown that for any compact subset 0, C Vi, 

E,rco.V(^r) :^(r?)rni^cF(z(if)5o-'LinCo-^)(Linr)An} ^ 

hmsup — '- — — > 0, 

for some finite subset Aq C F. Hence for some 7r2 G F, 

EircoJ^i^^)--MC)'^nncvoiZiH)g^'L,nG^'){L,nT)jn} ^ 
hmsup — ^ — -— > 0, 

By the remark following Theorem 15.11 Li n F is a Zariski dense lattice in Lj. 

Hence the sequence {Om} is focused, yielding contradiction. This finishes the proof. 
Proof of Theorem II .21 Without loss of generality we may assume that G is connected. Fix 
any F G Ay preserving each Om- If (|1.4j) does not hold, then {Om} is focused by Theorem 
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Since is a maximal comiected closed subgroup of G, it follows that there exists 
g £ G such that for any compact subset Q C Vi 

E^rcoJ^m ■ Pr(Or nnc v,Z{H^)H^gT} 
hmsup^ ■ — — > 0. 

We claim that 

Egrca^V(gr) :pr(g)rnz;oZ(ff°)gOgr^0} ^ ^ 

(5.10) limsup — ^ — — r > 

m— »oo '-'-'(, L^m j 

Suppose not. Then it follows that for any compact subset Q. of Vi, 

hmsup — — > 0. 

m— >oo ^[y^m) 

This is equivalent to saying that for any compact subset C of r\G, 

E,rci)^{^W:r\rg,gOnc = 0} 

hmsup — — — > 0. 

Hence there exists a sequence rjm G Om such that for any compact subset C C X, there exists 
m such that T\Tg^^H^ n C = 0. On the other hand, since is a proper maximal closed 
subgroup of G^, by Proposition 13.41 any weak limit of {fjy^r} in ViX U {oo}) is supported 
on X. This is a contradiction by Proposition 13.21 Hence is proved. 

It is easy to check that pr(^)r n voZ{H^)H^gT ^ implies 7i?g7~^ = gHg^^ for some 
7 G r. Hence (|5.1()|) implies: 

EfrcO™{'^(^r) • ^^a"^ = gHg-^ for some 7 e T} 
limsup ^ — — > 

m— >oo '-^\}-^'m) 

It follows that there exists ^0 G Cmo for some tuq such that 

v E5rcO„{^(^r) : 7/^57-1 = if^o for some 7 G P} 
lim sup — ^ > 

Observe that the condition ■jH^j^^ = H^^^ implies that g^^jg^^ G N{H), and the condition 
9^0^19^ ^ H imphes that pr(^)P = pr(^o)r. 

Since H has a finite index in the normalizer N{H), it follows that 

(5.11) hmsup — ^ — — > 0. 

m— >oo ^{^rnj 

Note that if pr(,^)P = pr(^o)r) then ti;(^r) = a;(^or). Since there can be at most one 
P-orbit ^P in Om such that pr(,^)P = pr(^o)r) (l^.llj) implies that 

a;(^or) • limsup — — — - > or equivalently liminf cj(C'm) < 00. 

m-*oo iO[Om) m^oo 

By Lemma ESI this contradicts the assumption on {Om}- Hence the proof is now complete. 
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6. Examples 

Theorem 6.1. Let Q be an integral quadratic form of signature (r, s) where r + s>4, r>2 
and s > 1. For any compact subset J7 of Vi with boundary of measure 0, 

Nm{Q, il) ~ • vol (0,) 

as m ^ oo along the fundamental discriminants. 

Proof. Consider the standard representation of the orthogonal group 0{Q) on V := W^~^^. 
Let Vm be the level set {x £ V : Q{x) = m}. By Witt's theorem the orthogonal group 0(Q) 
acts transitively on each Vi. The stabilizer of a vector vq in Vi is isomorphic to 0(r — 1, s). 
Note that the assumptions on the size of the parameters r and s guarantee that 0(r — 1, s) is 
non-compact and simple. It is well known that 0(r — 1, s)^ is a maximal connected subgroup 
of 0(r, s)*^. If we set F := 0{Q) Pi SLr+s(Z), then F E Ay. Under the assumption that 
r + s > 4 and rs = 1, ^ for all fundamental discriminants m (see [Oh]). Hence 

we may take Om = VmC^) to apply theorem 11.21 To check the condition (|1.3j) . note that if 
m ^ k are fundamental discriminants, then ^/rn^'^Vm{'L) H Vfc(Z) = 0; otherwise this 

would imply that ^{rnjlz) G Q, which can be seen to be false by an easy computation. 
Therefore Theorem 11.21 implies the claim. □ 

Let V := \X S M2„(M) : X* = —X} be the space of skew-symmetric matrices, so that the 
Pfaffian on V is defined by 

Pff2(X) = Det(X) and Pff(t;o) = 1 

where 

■ In 



-/„, 

E.g., for n = 2, we have Pff(xi, • • • ,X6) = x\X2 — + x^x%. 

Theorem 6.2. Let n >2. For any compact subset Q of Vi with boundary of measure 0, 

Nm(Pf^, ^) ~m-*oo " Vol (17). 

Proof. Consider the representation p : SL2n(IR) GL(y) defined by 

p{A){X) =A^XA 

where A G SL2n(lR) and X €z V. It is well known that SL2n(IR) acts transitively on Vi = {X E 
V : Pff(X) = 1}, so that Vm = m^^"'voSL2nO^)- The stabilizer H of vq is the symplectic 
group Sp2„(M) corresponding to vq, which is a maximal connected closed subgroup of SL2n( 
Clearly VmC^) / and SL2n(Z) preserves Vm(^) for each m. 

To check the condition 11.31 of Theorem ll.21 suppose for a given mo G N that pr(ymo(^)) 
pr(Hn(Z)). Then 

/ m \ 

In-l 

-m 

V -In^i J 

Hence m"^/" = mg ^^""k for some A; G N. This leads to m = mgA;"" < mo- Hence 
|{m G N : pr(y„o(Z)) = pr(y^(Z))}| < mo < oo. 



m-i/" 



G m-^/"y(Z) 
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Hence the claim follows from Theorem 11.21 □ 

Since the explicit representatives of SL2r!,(^)-orbits on can easily be written down in 

the Pff case, we can compute iOm, using the local density formula given in [GY]. For instance, 
for a prime p, 

2n-2 
1=0 

for some constant C independent of p. 

Theorem 6.3. Let n > 2. For any compact subset $7 of Vi = SL„(]R) with boundary of 
measure 0, 

Nmi^et, i7) ~m-^oo ' Vol (17). 

Proof. Consider the representation SL„(M) x SL„(M) on the space V = M„(M) given by 
X{A,B) = AXB-^ where X & V and A,B ^ SL„(M). The stabilizer of /„ is given by 
the diagonal embedding of SL„(M) in the product SL„(M) x SL„(M), which is a maximal 
connected closed subgroup. Clearly VmC^) 7^ and SL„(Z) x SL„(Z) preserves V{Z). The 
condition 11.31 can be checked similarly as in the case of Pfaffian. Hence our claim Theorem 
01 □ 

For the determinant case, the constant ujm is well known from the theory of Hecke opera- 
tors, (cf. [Sa], or [GO]). For instance, for any fixed /c S N and a prime p, 
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